Papers with human evaluator
Leveraging syntactic parsing to improve event annotation matching (D19-59)
Copied to clipboard
| Challenge: | Evaluating annotator consistency is crucial when building datasets for mention detection. |
| Approach: | They propose to use different fuzzy-matching functions to resolve this ambiguity by extracting syntactic heads present in annotations and using the Dice coefficient to measure similarity between sets. |
| Outcome: | The proposed functions are tested against the judgment of a human evaluator and show that the best-performing function agrees with the human . |
Early-Exit and Instant Confidence Translation Quality Estimation (2026.eacl-long)
Copied to clipboard
| Challenge: | Quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines. |
| Approach: | They propose an uncertainty-aware quality estimation model that matches previous approaches at a fraction of their costs. |
| Outcome: | The proposed method reduces evaluation costs by 50% and improves reranking performance. |
A Bayesian Topic Model for Human-Evaluated Interpretability (2022.lrec-1)
Copied to clipboard
| Challenge: | Topic modeling is an effective way to analyze unstructured textual data. |
| Approach: | They propose to combine nonparametric and weakly-supervised topic models to produce interpretable topics. |
| Outcome: | The proposed model outperforms weakly-supervised models in the field of topic modeling. |